Issues Related to Data Analysis and Quantitative Methods in PER

On the very same quiz, Question #8 asks the students to choose a vector diagram that most closely represents the gravitational forces that the earth and moon exert on each other. The three most popular choices are shown in the figure below. There are a number of issues that always arise, implicitly or explicitly, when conducting quantitative research and carrying out data analysis in Physics Education Research. (Most are relevant for qualitative research as well.) I. Validity. Broadly speaking, validity refers to the degree to which the conclusions of an investigation truthfully and accurately respond to some specific research questions. Among the particular issues that may arise is: Does your instrument provide data that could actually answer your research question? A common flaw is that the instrument (or test item) is not sufficiently focused, in this sense: To try to answer the question, “Do students understand concept A?” the test item (or test instrument) requires knowledge of concepts A, B, and C. Here, B and/or C might correspond to specific mathematical tools or formal representations. A related question that might arise is: Is your interpretation of the data an accurate representation of students’ knowledge? The correct answer “b” was given by 612% of students. In each of the five independent administrations of the quiz, the proportion of correct responses on Question #8 was about half that on Question #1 (0.43, 0.60, 0.59, 0.50, and 0.50). The implication seems to be that Question #8 was measuring not only students’ knowledge of Newton’s third law of motion and law of gravitation, but also (in part) students’ understanding of vector diagrams. This conclusion is considerably strengthened by the fact that 3447% of students gave answer “c” on Question #8 [answer “a”: 43-55%]. The “c” response corresponds to the force exerted by the more massive object having the smaller magnitude, a response that was given by only 3-6% of the same students on Question #1. We see, then, that the validity of two inferences that might have been drawn from the results on Question #8 are thrown into question: (1) the proportion of students who misunderstood Newton’s third law, and (2) the proportion who believed that in a gravitational interaction involving two masses, the more massive object exerts the smaller magnitude force. Although a more definitive analysis of students’ reasoning on these questions must await examination of interview data (currently underway), it seems clear that the validity of conclusions that might have been based on only one of these test items would be very uncertain. M E